Apparatus for editing image using depth map and method thereof

ABSTRACT

An electronic device is provided. The electronic device includes a display, a memory configured to store instructions, and a processor, wherein the processor is configured to execute the stored instructions to display an image and a user interface (UI) representing depth information of the image through the display in response to a first user input for editing the image, receive a second user input for selecting an object to be added in the image, display at least a part of the object in the image based on a depth value of the object and depth information of a first region in which the object is located in the image, and display an indicator representing the depth value of the object in the UI.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of prior application Ser.No. 16/059,586, filed on Aug. 9, 2018, which application is based on andclaims priority under 35 U.S.C. § 119(a) of a Korean patent applicationnumber 10-2017-0104774, filed on Aug. 18, 2017, in the KoreanIntellectual Property Office, the disclosure of which is incorporated byreference herein its entirety.

BACKGROUND 1. Field

The disclosure relates to an apparatus for editing an image using adepth map and a method thereof.

2. Description of Related Art

An electronic device may obtain a color image and a depth map using atleast one image sensor. The color image may include a red-green-blue(RGB) value of each pixel included therein and the depth map may includea depth value of each pixel. The depth value may be a valuecorresponding to a distance between a subject and an electronic device(or at least one image sensor). For example, when the depth values inone region including a plurality of pixels are greater than those inanother region in a depth map, this may mean that the subjectcorresponding to the one region is closer to the electronic device thanthe subject in the other region.

The electronic device may provide various user interfaces (UI) to enablea user of the electronic device to edit an image. For example, the imageediting may include image composition or inserting of an object (e.g., asticker, a label, a drawing, or the like) in an image.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providean electronic device for editing an image using a depth map and a methodthereof.

Another aspect of the disclosure is to provide an electronic device mayprovide a UI for editing an image to a user, but does not provide afunction of editing an image using a depth map.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, an electronic device isprovided. The electronic device includes a display, a memory configuredto store instructions, and a processor coupled to the display and thememory, wherein the processor is configured to execute the storedinstructions to display an image and a user interface (UI) representingdepth information of the image through the display in response to afirst user input for editing the image, receive a second user input forselecting an object to be added in the image, display at least a part ofthe object in the image based on a depth value of the object and depthinformation of a first region in which the object is located in theimage, and display an indicator representing the depth value of theobject in the UI.

In accordance with another aspect of the disclosure, a method of anelectronic device is provided. The method includes displaying an imageand a UI representing depth information of the image in response to afirst user input for editing the image, receiving a second user inputfor selecting an object to be added in the image, displaying at least apart of the object in the image based on a depth value of the object anddepth information of a first region in which the object is located inthe image, and displaying an indicator representing the depth value ofthe object in the UI.

In accordance with an aspect of the disclosure, an electronic device isprovided. The electronic device includes at least one image sensor, adisplay, a memory configured to store instructions, and a processor,wherein the processor is configured to execute the stored instructionsto obtain a color image and a depth map corresponding to the color imagethrough the at least one image sensor, generate an image by combiningthe color image with the depth map, display the image and a UIrepresenting depth information of the image through the display inresponse to a first user input for editing the image, receive a seconduser input for selecting an object to be added in the image, display atleast a part of the object in the image based on a depth value of theobject and depth information of a first region in which the object islocated in the image, and display an indicator representing the depthvalue of the object in the UI.

According to the embodiments of the disclosure, the electronic devicemay provide a function of editing an image using an UI showing depthinformation of an image.

According to the embodiments of the disclosure, the electronic devicemay provide a function of displaying an object of which a depth value ischanged in response to reception of various user inputs.

In addition, various effects that are directly or indirectly understoodthrough the disclosure may be provided.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a block diagram of an electronic device in a networkenvironment according to an embodiment of the disclosure;

FIG. 2 is a block diagram illustrating a camera module according to anembodiment of the disclosure;

FIG. 3 is a flowchart illustrating an operation of an electronic devicefor editing an image using a UI showing depth information of an imageaccording to an embodiment of the disclosure of the disclosure;

FIG. 4 is a flowchart illustrating an operation of an electronic devicefor editing an image using a user interface (UI) showing depthinformation of an image and an indicator representing a depth value ofan object according to an embodiment of the disclosure;

FIG. 5 is a view illustrating a UI showing depth information of an imageaccording to an embodiment of the disclosure;

FIG. 6A is a view illustrating a sequence of operations of editing animage using a UI showing depth information of an image according to anembodiment of the disclosure;

FIG. 6B is a view illustrating a sequence of operations of editing animage using another UI showing depth information of an image accordingto an embodiment of the disclosure;

FIG. 7 is a view illustrating a sequence of operations of releasing anindicator representing the edited state of an object according to anembodiment of the disclosure;

FIG. 8 is a view illustrating a sequence of operations of combiningbackground images based on depth information of an image according to anembodiment of the disclosure;

FIG. 9 is a flowchart illustrating an operation of an electronic devicefor editing a size of an object based on a modified depth valueaccording to an embodiment of the disclosure;

FIG. 10 is a view illustrating a principle of an operation of editing asize of an object based on a modified depth value according to anembodiment of the disclosure;

FIG. 11 is a view illustrating a sequence of operations of editing asize of an object based on a modified depth value according to anembodiment of the disclosure;

FIG. 12 is a view illustrating a sequence of operations of editing aplurality of objects in an image by using a UI representing depthinformation of the image according to an embodiment of the disclosure;

FIG. 13 is a flowchart illustrating an operation of an electronic devicefor selecting a subject in an image by using a UI representing depthinformation of the image according to an embodiment of the disclosure;

FIG. 14 is a flowchart illustrating a sequence of operations ofselecting a subject in an image by using a UI representing depthinformation of the image according to an embodiment of the disclosure;

FIG. 15A is a view illustrating an image composition using a UIrepresenting depth information of an image according to an embodiment ofthe disclosure;

FIG. 15B is a view illustrating an image composition using a UIrepresenting depth information of an image according to an embodiment ofthe disclosure;

FIG. 16 is a flowchart illustrating an electronic device for displayingan object of which a depth value is modified based on a user inputaccording to an embodiment of the disclosure; and

FIGS. 17A, 17B, and 17C are views illustrating a sequence of operationsof displaying an object of which a depth value is modified based on auser input according to various embodiments of the disclosure.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

DETAILED DESCRIPTION

The following description with reference to accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

By the term “substantially” it is meant that the recited characteristic,parameter, or value need not be achieved exactly, but that deviations orvariations, including for example, tolerances, measurement error,measurement accuracy limitations and other factors known to those ofskill in the art, may occur in amounts that do not preclude the effectthe characteristic was intended to provide.

FIG. 1 is a block diagram of an electronic device in a networkenvironment according to an embodiment of the disclosure.

Referring to FIG. 1, a first electronic device 101 may communicate witha second electronic device 102 through a first network 198 (e.g., ashort-range wireless communication) or may communicate with a thirdelectronic device 104 or a server 108 through a second network 199(e.g., a long-distance wireless communication) in a network environment100. According to an embodiment of the disclosure, the first electronicdevice 101 may communicate with the third electronic device 104 throughthe server 108. According to an embodiment of the disclosure, the firstelectronic device 101 may include a processor 120, a memory 130, aninput device 150, a sound output device 155, a display device 160, anaudio module 170, a sensor module 176, an interface 177, a haptic module179, a camera module 180, a power management module 188, a battery 189,a communication module 190, a subscriber identification module 196, andan antenna module 197. According to some embodiments of the disclosure,at least one (e.g., the display device 160 or the camera module 180)among components of the first electronic device 101 may be omitted orother components may be added to the first electronic device 101.According to some embodiments of the disclosure, some components may beintegrated and implemented as in the case of the sensor module 176(e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor)embedded in the display device 160 (e.g., a display or a display unit).

The processor 120 may operate, for example, software (e.g., a program140) to control at least one of other components (e.g., a hardware orsoftware component) of the first electronic device 101 connected to theprocessor 120 and may process and compute a variety of data. Theprocessor 120 may load a command set or data, which is received fromother components (e.g., the sensor module 176 or the communicationmodule 190), into a volatile memory 132, may process the loaded commandor data, and may store result data into a nonvolatile memory 134.According to an embodiment of the disclosure, the processor 120 mayinclude a main processor 121 (e.g., a central processing unit or anapplication processor) and an auxiliary processor 123 (e.g., a graphicprocessing device, an image signal processor, a sensor hub processor, ora communication processor), which operates independently from the mainprocessor 121, additionally or alternatively uses less power than themain processor 121, or is specified to a designated function. In thiscase, the auxiliary processor 123 may operate separately from the mainprocessor 121 or embedded.

The auxiliary processor 123 may control, for example, at least some offunctions or states associated with at least one component (e.g., thedisplay device 160, the sensor module 176, or the communication module190) among the components of the first electronic device 101 instead ofthe main processor 121 while the main processor 121 is in an inactive(e.g., sleep) state or together with the main processor 121 while themain processor 121 is in an active (e.g., an application execution)state. According to an embodiment of the disclosure, the auxiliaryprocessor 123 (e.g., the image signal processor or the communicationprocessor) may be implemented as a part of another component (e.g., thecamera module 180 or the communication module 190) that is functionallyrelated to the auxiliary processor 123. The memory 130 may store avariety of data used by at least one component (e.g., the processor 120or the sensor module 176) of the first electronic device 101, forexample, software (e.g., the program 140) and input data or output datawith respect to commands associated with the software. The memory 130may include the volatile memory 132 or the nonvolatile memory 134.

The program 140 may be stored in the memory 130 as software and mayinclude, for example, an operating system 142, a middleware 144, or anapplication 146.

The input device 150 may be a device for receiving a command or data,which is used for a component (e.g., the processor 120) of the firstelectronic device 101, from an outside (e.g., a user) of the firstelectronic device 101 and may include, for example, a microphone, amouse, or a keyboard.

The sound output device 155 may be a device for outputting a soundsignal to the outside of the first electronic device 101 and mayinclude, for example, a speaker used for general purposes, such asmultimedia play or recordings play, and a receiver used only forreceiving calls. According to an embodiment of the disclosure, thereceiver and the speaker may be either integrally or separatelyimplemented.

The display device 160 may be a device for visually presentinginformation to the user and may include, for example, a display, ahologram device, or a projector and a control circuit for controlling acorresponding device. According to an embodiment of the disclosure, thedisplay device 160 may include a touch circuitry or a pressure sensorfor measuring an intensity of pressure on the touch.

According to various embodiments of the disclosure, when the displaydevice 160 includes a touch circuit or a pressure sensor, the displaydevice 160 may receive various types of user inputs. According to anembodiment of the disclosure, a user input may include an input using atleast one body (e.g., a finger) of a user. For example, the input usingat least one body of a user may include an operation of touching a touchscreen at least once (which may be referred to as a tap operation), anoperation of touching a touch screen by at least one body of a user fora predetermined time period (which may be referred to as atouch-and-hold or press operation), an operation of touching a touchscreen by at least one body of a user at a predetermined pressure orhigher (which may be referred to as a pressure touch operation), anoperation of moving at least one body of a user without restriction intime and direction while the at least one body touches a touch screen(which may be referred to as a panning operation), an operation ofmoving at least one body of a user without restriction in time anddirection while the at least one body touches a touch screen (which maybe referred to as a panning operation), an operation of moving at leastone body of a user faster than a swipe operation (which may be referredto as a flick operation), an operation of moving at least one body of auser while the at least one body touches an object displayed on a touchscreen (which may be referred to as a drag operation), an operation ofnarrowing or widening an interval between at least two or more bodies ofa user while the at least two or more bodies touch the touch screen(which may be referred to as a pinch-in or pinch-out operation), anoperation of rotating at least one body of a user while the at least onebody touches a touch screen (which may be referred to as a rotatingoperation), and combinations thereof. According to another embodiment ofthe disclosure, a user input may include an input using an externaldevice. For example, the external device may include an electronic pen(which may be referred to as a touch pen). For example, the input usingan external device may include a tap operation, a press operation, apressure touch operation, a hovering operation, a panning operation, aswipe operation, a flick operation, a drag operation, a rotatingoperation, a button input operation (which may include, for example, ascheme of pushing a button included in an electronic pen in a touch orhovering state), and combinations thereof.

The audio module 170 may convert a sound and an electrical signal indual directions. According to an embodiment of the disclosure, the audiomodule 170 may obtain the sound through the input device 150 or mayoutput the sound through an external electronic device (e.g., the secondelectronic device 102 (e.g., a speaker or a headphone)) wired orwirelessly connected to the sound output device 155 or the firstelectronic device 101.

The sensor module 176 may generate an electrical signal or a data valuecorresponding to an operating state (e.g., power or temperature) insideor an environmental state outside the first electronic device 101. Thesensor module 176 may include, for example, a gesture sensor, a gyrosensor, a barometric pressure sensor, a magnetic sensor, an accelerationsensor, a grip sensor, a proximity sensor, a color sensor, an infraredsensor, a biometric sensor, a temperature sensor, a humidity sensor, oran illuminance sensor.

The interface 177 may support a designated protocol wired or wirelesslyconnected to the external electronic device (e.g., the second electronicdevice 102). According to an embodiment of the disclosure, the interface177 may include, for example, a high-definition multimedia interface(HDMI), a universal serial bus (USB) interface, a secure digital (SD)card interface, or an audio interface.

A connection terminal 178 may include a connector that physicallyconnects the first electronic device 101 to the external electronicdevice (e.g., the second electronic device 102), for example, an HDMIconnector, a USB connector, an SD card connector, or an audio connector(e.g., a headphone connector).

The haptic module 179 may convert an electrical signal to a mechanicalstimulation (e.g., vibration or movement) or an electrical stimulationperceived by the user through tactile or kinesthetic sensations. Thehaptic module 179 may include, for example, a motor, a piezoelectricelement, or an electric stimulator.

The camera module 180 may shoot a still image or a video image.According to an embodiment of the disclosure, the camera module 180 mayinclude, for example, at least one lens, an image sensor, an imagesignal processor, or a flash.

The power management module 188 may be a module for managing powersupplied to the first electronic device 101 and may serve as at least apart of a power management integrated circuit (PMIC).

The battery 189 may be a device for supplying power to at least onecomponent of the first electronic device 101 and may include, forexample, a non-rechargeable (primary) battery, a rechargeable(secondary) battery, or a fuel cell.

The communication module 190 may establish a wired or wirelesscommunication channel between the first electronic device 101 and theexternal electronic device (e.g., the second electronic device 102, thethird electronic device 104, or the server 108) and supportcommunication execution through the established communication channel.The communication module 190 may include at least one communicationprocessor operating independently from the processor 120 (e.g., theapplication processor) and supporting the wired communication or thewireless communication. According to an embodiment of the disclosure,the communication module 190 may include a wireless communication module192 (e.g., a cellular communication module, a short-range wirelesscommunication module, or a global navigation satellite system (GNSS)communication module) or a wired communication module 194 (e.g., anlocal area network (LAN) communication module or a power linecommunication module) and may communicate with the external electronicdevice using a corresponding communication module among them through thefirst network 198 (e.g., the short-range communication network, such asa Bluetooth, a WiFi direct, or an infrared data association (IrDA)) orthe second network 199 (e.g., the long-distance wireless communicationnetwork, such as a cellular network, an internet, or a computer network(e.g., LAN or wide area network (WAN)). The above-mentioned variouscommunication modules 190 may be implemented into one chip or intoseparate chips, respectively.

According to an embodiment of the disclosure, the wireless communicationmodule 192 may identify and authenticate the first electronic device 101using user information stored in the subscriber identification module196 in the communication network.

The antenna module 197 may include one or more antennas to transmit orreceive the signal or power to or from an external source. According toan embodiment of the disclosure, the communication module 190 (e.g., thewireless communication module 192) may transmit or receive the signal toor from the external electronic device through the antenna suitable forthe communication method.

Some components among the components may be connected to each otherthrough a communication method (e.g., a bus, a general purposeinput/output (GPIO), a serial peripheral interface (SPI), or a mobileindustry processor interface (MIPI)) used between peripheral devices toexchange signals (e.g., a command or data) with each other.

According to an embodiment of the disclosure, the command or data may betransmitted or received between the first electronic device 101 and theexternal third electronic device 104 through the server 108 connected tothe second network 199. Each of the second electronic device 102 and thethird electronic device 104 may be the same or different types as orfrom the first electronic device 101. According to an embodiment of thedisclosure, all or some of the operations performed by the firstelectronic device 101 may be performed by another electronic device or aplurality of external electronic devices. When the first electronicdevice 101 performs some functions or services automatically or byrequest, the first electronic device 101 may request the externalelectronic device to perform at least some of the functions related tothe functions or services, in addition to or instead of performing thefunctions or services by itself. The external electronic devicereceiving the request may carry out the requested function or theadditional function and transmit the result to the first electronicdevice 101. The first electronic device 101 may provide the requestedfunctions or services based on the received result as is or afteradditionally processing the received result. To this end, for example, acloud computing, distributed computing, or client-server computingtechnology may be used.

FIG. 2 is a block diagram illustrating a camera module according to anembodiment of the disclosure.

Referring to FIG. 2, the camera module 180 may include a lens assembly210, a flash 220, an image sensor 230, an image stabilizer 240, a memory(e.g., a buffer memory) 250, or an image signal processor 260. The lensassembly 210 may collect light emitted from a subject which is an objectof image capturing. The lens assembly 210 may include one or morelenses. According to an embodiment of the disclosure, the camera module180 may include a plurality of lens assemblies 210. In this case, thecamera module 180 may be, for example, a dual camera, a 360-degreecamera, or a spherical camera. The plurality of lens assemblies 210 mayhave the same lens properties (e.g., angle of view, focal length,autofocus, f number, or optical zoom), or at least one lens assembly mayhave a property different from that of another lens assembly. Forexample, the lens assembly 210 may include a wide-angle lens or atelephoto lens. The flash 220 may emit a light source used to enhancethe light emitted from the subject. The flash 220 may include one ormore light emitting diodes (e.g., RGB light emitting diode (LED), awhite LED, an infrared LED, or an ultraviolet LED), or a xenon lamp.

The image sensor 230 may acquire an image corresponding to the subjectby converting the light transmitted from the subject through the lensassembly 210 into an electrical signal. According to an embodiment ofthe disclosure, the image sensor 230 may include, for example, oneselected from image sensors having different properties, such as an RGBsensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, aplurality of image sensors having the same property, or a plurality ofimage sensors having different properties. For example, each imagesensor included in the image sensor 230 may be implemented with acharged coupled device (CCD) sensor or a complementary metal oxidesemiconductor (CMOS) sensor. According to an embodiment of thedisclosure, when the camera module 180 includes the plurality of lensassemblies 210, the image sensor 230 includes a plurality of imagesensors that acquire images from each of the plurality of lensassemblies 210. For example, one of the image sensors may acquire acolor image and another image sensor may acquire a depth map (which maybe referred to as a depth map image). As another example, each of theplurality of image sensors may cooperate with each another to obtain acolor image and a depth map.

The image stabilizer 240 may move or control (e.g., adjustment ofread-out timing, and the like) at least one lens included in the lensassembly 210 or the image sensor 230 in a specific direction to at leastpartially compensate for a negative effect (e.g., image blurring) due tomovement of the camera module 180 or the first electronic device 101including the camera module 180 with respect to the image beingphotographed in response to the movement of the camera module 180 or thefirst electronic device 101. According to an embodiment of thedisclosure, the image stabilizer 240 may be implemented, for example,with an optical image stabilizer and may detect the movement by using agyro sensor (not shown) or an acceleration sensor (not shown) locatedinside or outside the camera module 180.

The memory 250 may at least temporarily store at least a part of theimage obtained through the image sensor 230 for a next image processingoperation. For example, when image acquisition by the shutter is delayedor a plurality of images are acquired at a high speed, the acquiredoriginal image (e.g., a high-resolution image) is stored in the memory250, and a copy image (e.g., a low-resolution image) corresponding tothe original image may be previewed through the display device 160.Then, when a specified condition (e.g., a user input or systeminstruction) is met, at least a part of the original image stored in thememory 250 may be obtained and processed, for example, by the imagesignal processor 260. According to an embodiment of the disclosure, thememory 250 may be implemented with at least a part of the memory 130 ora separate memory operated independent of the memory 130.

The image signal processor 260 may perform image processing (e.g., depthmap generation, three-dimensional modeling, panorama generation, featurepoint extraction, image composition, or image compensation (e.g., noisereduction, resolution adjustment, brightness adjustment, blurring,sharpening, or softening)) of an image acquired through the image sensor230 or an image stored in the memory 250. Additionally or alternatively,the image signal processor 260 may perform control (e.g., exposure timecontrol, lead-out timing control, or the like) of at least one of thecomponents (e.g., the image sensor 230) included in the camera module180. The image processed by the image signal processor 260 may be againstored in the memory 250 for additional processing or may be transmittedto an external component of the camera module 180 (e.g., the memory 130,the display device 160, the second electronic device 102, the thirdelectronic device 104, or server 108). According to an embodiment of thedisclosure, the image signal processor 260 may be configured as at leasta part of the processor 120, or may be configured as a separateprocessor operating independently of the processor 120. When the imagesignal processor 260 is configured as a separate processor, the imagesprocessed by the image signal processor 260 may be displayed by theprocessor 120 through the display device 160 as they are or after beingadditionally image-processed.

According to an embodiment of the disclosure, the first electronicdevice 101 may include two or more camera modules 180 having differentproperties or functions. In this case, for example, the at least onecamera module 180 may be a wide-angle camera or a front camera and atleast one other camera module may be a telephoto camera or a rearcamera.

In the drawings described below, each operation may be implemented by anelectronic device (e.g., the first electronic device 101), or aprocessor (e.g., the processor 120 or the image signal processor 260)included in the electronic device. When each operation is implemented bythe processor, the processor may perform each operation by executinginstructions stored in the memory (e.g., the memory 130 or the memory250) of the electronic device.

FIG. 3 is a flowchart illustrating an operation of an electronic devicefor editing an image using a UI showing depth information of an imageaccording to an embodiment of the disclosure.

Referring to FIG. 3, in operation 310, the electronic device (e.g., thefirst electronic device 101) and the processor 120 or the image signalprocessor 260 may receive a user input for editing an image. Forexample, the electronic device may receive the user input for editing animage while displaying the image. As another example, the electronicdevice may receive a user input for editing one of a plurality of imageswhile displaying thumbnails of the images.

In operation 320, the electronic device may display a UI representing animage and depth information of the image in response to the reception ofthe user input for editing an image. The relationship of the UIrepresenting an image, a depth map and depth information will bedescribed in FIG. 5.

FIG. 5 is a view illustrating a UI showing depth information of an imageaccording to an embodiment of the disclosure.

Referring to FIG. 5, the electronic device may obtain a color image 500a and a depth map 500 b through at least one image sensor. The colorimage 500 a may include color information (e.g., RGB values) of eachpixel. The electronic device may display a background subject 510 and aforeground subject 520 by using the color information of each pixelincluded in the color image 500 a. The depth map 500 b may include depthvalues of each pixel. For example, when the electronic device supports amemory capacity of 1 byte per pixel, the depth value may be expressed asa value from 0 to 255. According to an embodiment of the disclosure, theelectronic device may be set such that the closer the distance betweenthe electronic device and the subject is, the greater the depth valueis, or the closer the distance between the electronic device and thesubject is, the less the depth value is. Various embodiments of thedisclosure described below describe an example in which the depth valueis increased as the distance between the electronic device and thesubject is closer. According to an embodiment of the disclosure, thedepth values of pixels may be different from each other, but the depthvalues of the plurality of pixels constituting a part of an image regionmay be the same or similar to each other. For example, the depth valuesof the pixels constituting the foreground subject 520 in the color image500 a may be the same or similar to each other, and the depth values ofthe pixels constituting the background subject 510 may be the same orsimilar to each other. In this case, the pixels constituting an imageregion 512 (i.e., the region corresponding to the background subject510) in the depth map 500 b may be represented at the same or similarbrightness, and an image region 522 (i.e., the region corresponding tothe foreground subject 520) may be represented at the same or similarbrightness.

According to an embodiment of the disclosure, the electronic device maygenerate a UI 505 representing the depth information of the image basedon the depth map 500 b. In various embodiments of the disclosure, animage may refer to an image generated by combining a color image and adepth map. In various embodiments of the disclosure, the depthinformation may include a depth value distribution of pixels.

FIG. 5 illustrates a horizontal bar graph type of UI 505, but the typeof UI 505 is not limited to that shown in FIG. 5, and may vary dependingon the implementation scheme. For example, the UI 505 may be a verticalbar graph. According to an embodiment of the disclosure, the horizontalaxis of the UI 505 may correspond to a depth value range (e.g., from 0to 255) of the depth map 500 b, or may correspond to a value obtained byconverting the depth value range into the range of 0 to 100. Accordingto an embodiment of the disclosure, the UI 505 may represent depthinformation of at least a region in the depth map 500 b. For example,the depth values of the pixels corresponding to the image region 522 inthe depth map 500 b may be distributed in a specific range, so that theelectronic device may display a depth range 525 representing the depthinformation of the image region 522 in the UI 505. In the sameprinciple, the electronic device may display a depth range 515 thatrepresents the depth information of the image region 512 in the UI 505.

Referring back to FIG. 3, in operation 330, the electronic device maydisplay at least a part of the object in the image based on the depthvalue of the object being added in the image and the depth informationof the image. For example, the object may include a sticker, a label, adrawing, a memo, an image, a mosaic, or calligraphy. According to anembodiment of the disclosure, the electronic device may display at leasta part of the object in the image by comparing the depth value of theobject with the depth information of the region (which may be referredto as a first region) in which the object is displayed in the image. Forexample, referring to FIG. 5, the object may be inserted at a locationat which a part of the background subject 510 overlaps a part of theforeground subject 520. The electronic device may compare the depthvalue of the object, the depth information of the image region 512 andthe depth information of the image region 522 with one another. When thedepth value of the object is greater than the maximum depth value of theimage region 522, the electronic device may display the entire object inthe image. In this case, the object may be seen to the user as beingarranged forward in the image. When the depth value of the object isless than the minimum depth value of the image region 512, theelectronic device may not display the object in the image. When thedepth value of the object is less than the minimum depth value of theimage region 522 and greater than the maximum depth value of the imageregion 512, the electronic device may display only a part of the objectthat overlaps the image region 512. In this case, the object may bearranged behind the foreground subject 520 in the image, and may be seento the user as being arranged in front of the background subject 510.

In operation 340, the electronic device may display, in the UI, anindicator that indicates the depth value of the object. According to anembodiment of the disclosure, the electronic device may display theindicator in response to receiving a user input for selecting an object.The indicator may be displayed in the UI based on the relationshipbetween the depth value of the object and the depth information of theimage determined in operation 330. For example, based on whether thedepth value of the object is greater than the depth value of each of thepixels corresponding to the first region, the indicator may be arrangedleft (or top) or right (or bottom) in the UI.

The electronic device may provide a function of adjusting the depth ofan object in the image by displaying a UI representing the depth valueof the object and the depth information of the image. A specificembodiment for adjusting the depth of an object will be described inFIG. 4.

FIGS. 4, 6A and 6B are views illustrating an operation of an electronicdevice for editing an image using a UI showing depth information of animage and an indicator representing a depth value of an object accordingto various embodiments of the disclosure. FIG. 4 may illustrate morespecifically implemented operations of operations 330 and 340 of FIG. 3.

Referring to FIG. 4, in operation 410, the electronic device (e.g., thefirst electronic device 101, the processor 120, or the image signalprocessor 260) may receive a user input for selecting an object whilethe UI representing the image and the depth information of the image isdisplayed. The user input for selecting an object may include at leastone of operations of selecting one of various types of objects (e.g., asticker, a label, a drawing, a memo, an image, a mosaic, or calligraphy)and arranging the selected object in the first region.

Referring to FIG. 6A, the electronic device may display an image 600 aincluding the background and foreground subjects 510 and 520 and the UI505 representing the depth information of the image 600 a. For example,since the depth values of the pixels constituting the background subject510 are the same or similar to each other and the depth values of thepixels constituting the foreground subject 520 may be the same orsimilar to each other, the UI 505 may include the depth range 515representing the depth information of the image region corresponding tothe background subject 510 and the depth range 525 representing thedepth information of the image region corresponding to the foregroundsubject 520. Although FIG. 6A illustrates the UI 505 in a horizontalform, which is arranged in a lower end area in the image 600 a, the formand location of the UI 505 is not limited to those illustrated in FIG.6A, and may be variously implemented depending on the implementationschemes. For example, the UI 505 may be arranged horizontally at anupper end in the image 600 a, or vertically at the left or right side inthe image. As another example, the UI 505 may be arranged horizontallyon a lower or an upper screen than the image 600 a, or vertically on theleft or right screen rather than the image 600 a. According to anembodiment of the disclosure, the UI 505 may be displayed in atransparent form. According to an embodiment of the disclosure, thelightness of each of the depth ranges 515 and 525 may be substantiallyequal to or different from each other.

In operation 420, in response to the user input for selecting an object,the electronic device may display the object having a pre-determineddepth value and the indicator representing the depth value of theobject. For example, the predetermined depth value may be the maximumdepth value of the image. In this case, the electronic device candisplay the entire object in the image. The object may be seen to theuser as being arranged ahead of other subjects. The electronic devicemay display the indicator representing the depth of the object to bearranged at the leftmost (or rightmost) position in the UI. As anotherexample, the predetermined depth value may be the minimum depth value ofthe image. In this case, the electronic device may not display theobject in the image. The object may be seen to the user as beingarranged behind other objects. The electronic device may display theindicator representing the depth of the object to be arranged at therightmost (or leftmost) position in the UI.

Referring to FIG. 6A, the electronic device may display an object 530and an indicator 535 representing the depth of the object 530 inresponse to receiving a user input 610 for selecting an object while theimage 600 a is displayed. According to an embodiment of the disclosure,the electronic device may further display an indicator 540 representingthe state that the object 530 is currently being edited. According to anembodiment of the disclosure, the electronic device may set a default(i.e., predetermined) depth value of the object 530 to the maximum depthvalue of the image 600 a. In this case, the object 530 may be seen tothe user to be arranged most forward (i.e., forward of the foregroundsubject 520) in the image 600 a. The electronic device may display theindicator 535 such that the indicator 535 is arranged at the leftmostside in the UI 505. Although FIG. 6A illustrates the indicator 535arranged at the left side in proportion to the magnitude of the depthvalue, the indicator 535 may be set to be arranged at the right side inproportion to the magnitude of the depth value according to an implementscheme. The color or brightness of the indicator 535 may besubstantially the same as or different from the color or brightness ofeach of the depth ranges 515 and 525.

In operation 430, the electronic device may receive a user input formoving the displayed indicator. For example, referring to FIG. 6A, theelectronic device may receive a user input 620 for moving the indicator535. For example, the user input 620 may include a tap operation, apress operation, a pressure touch operation, a swipe operation, a flickoperation, or a drag operation.

In operation 440, the electronic device may modify the depth value ofthe object based on the indicator moved by the user input. For example,the electronic device may increase or decrease the depth value of theobject based on the direction and travel distance of the movedindicator.

In operation 450, the electronic device may display at least a part ofthe object in the image based on the modified depth value of the objectand the depth information of the image. The electronic device maydisplay at least a part of the object in the image by comparing themodified depth value of the object with the depth information of theregion (i.e., the first region) in which the object is displayed in theimage.

Referring to FIG. 6A, the electronic device may display at least a partof the object 530 having the modified depth value in response to theuser input 620. Since the indicator 535 has moved to the right (i.e.,the direction in which the depth value decreases), the electronic devicecan reduce the depth value of the object 530 based on the traveldistance of the indicator 535. When the reduced depth value of theobject 530 is smaller than the minimum depth value of the image areacorresponding to the foreground subject 520 and greater than the maximumdepth value of the image region corresponding to the background subject510 (in other words, when the indicator 535 is arranged between thedepth ranges 515 and 525 in the UI 505), the electronic device maydisplay the object 530 except for the region which overlaps theforeground subject 520 (where the object 530 may be seen to the user asbeing arranged in front of the background subject 510 and behind theforeground subject 520).

According to an embodiment of the disclosure, the electronic device mayprovide a user experience of adjusting the depth of the object using aUI in which the depth ranges of the subjects are not represented. Forexample, referring to FIG. 6B, the UI 505 may not include the depthranges 515 and 525 corresponding to the background subject 510 and theforeground subject 520, respectively. In other words, the electronicdevice only analyzes the color image 500 a and the depth map 500 b todetermine the depth range 525 and the depth range 515, and does notdisplay the determined depth ranges 515 and 525 through the display. Inthis case, the user of the electronic device may receive a visual effectin which the depth of the object 530 is modified using the relativeposition of the indicator 535 in the UI 505, without visually comparingthe position of the indicator 535 with the depth ranges 515 and 525.

FIG. 7 is a view illustrating a sequence of operations of releasing anindicator representing the edited state of an object according to anembodiment of the disclosure.

Referring to FIG. 7, while the object 530 and the indicator 540representing that the object 530 is in the edit state are displayed inthe image 600 a, the electronic device may receive a user input 710 forselecting a region other than the object 530 in the image 600 a. Theuser input 710 may include, for example, at least one or more tapoperations.

According to an embodiment of the disclosure, in response to the userinput 710, the electronic device may release the display of theindicator 540. Depending on the implement scheme, the electronic devicemay change the color of the indicator 535 in response to the user input710.

FIG. 8 is a view illustrating a sequence of operations of combiningbackground images based on depth information of an image according to anembodiment of the disclosure.

Referring to FIG. 8, the electronic device may receive a user input 810for selecting a background image. Although FIG. 8 illustrates an examplein which the electronic device receives the user input 810 while theobject 530 is displayed, the electronic device may receive the userinput 810 while the image 600 a into which the object 530 is notinserted is displayed. For example, the user input 810 may include atleast one or more tap operations.

According to an embodiment of the disclosure, in response to the userinput 810, the electronic device may combine a background image 600 bhaving the predetermined depth value with the image 600 a, and displayan indicator 825 representing the depth value of the background image600 b in the UI 505. For example, as illustrated in FIG. 8, thepredetermined depth value may be smaller than the depth value of theobject 530 and greater than the maximum depth value of the depthinformation corresponding to the depth range 515 (i.e., the indicator825 may be arranged between the indicator 535 and the depth range 515).In this case, the electronic device may display the background image 600b except for the region that overlaps the object 530 and the foregroundsubject 520. The background image 600 b may be seen to the user as beinglocated behind the foreground subject 520 and the object 530.

Although not illustrated in FIG. 8, the electronic device may displaythe background image 600 b of which the depth is modified, in responseto the user input for moving the indicator 825. For example, when theindicator 825 is moved to the indicator 535 by the user input, theelectronic device may display the background image 600 b except for theregion that overlaps the foreground subject 520. The background image600 b may be seen to the user as being located behind the foregroundsubject 520, and the object 530 may not be seen to the user.

The electronic device may provide a function of more easily combiningthe background image by adjusting the depth of the background imagecombined with the image using the UI representing the depth informationand the depth value.

FIGS. 9 to 11 are views illustrating a sequence of operations of editinga size of an object based on a modified depth value according to variousembodiments of the disclosure. The operations illustrated in FIG. 9 maybe operations that more specifically implement operations 440 and 450 ofFIG. 4.

Referring to FIG. 9, in operation 910, the electronic device maydetermine the size of the object based on the modified depth value ofthe object. For example, the electronic device may determine the size ofthe object based on the principle illustrated in FIG. 10.

Referring to FIG. 10, the distance between a depth 1010 and a user maybe shorter than the distance between a depth 1020 and a user. When it isassumed that the electronic device is set to increase the depth valueproportionally as the distance between the subject and the user (or theelectronic device) is closer, the depth value of the depth 1010 may begreater than that of the depth 1020. The electronic device may receivethe user input for modifying the depth value of an object 1015 havingthe depth value of the depth 1010 into the depth value of the depth1020. The depth value of the object 1015 may be reduced by referencenumeral 1030. The electronic device may modify the size (or area) of theobject 1015 into the size of an object 1025 in proportion to the depthvalue reduced by reference numeral 1030.

In operation 920, the electronic device may display the object of whichthe depth and size are modified, based on the modified depth value ofthe object, the size of the object and the depth information of theimage. For example, referring to FIG. 11, while the image 600 a, theobject 530, and the indicator 535 are displayed, the electronic devicemay receive a user input 1110 for moving the indicator 535 to the rightside. For example, the user input 1110 may include a tap operation, apress operation, a pressure touch operation, a swipe operation, a flickoperation, or a drag operation. In response to the user input 1110, theelectronic device may display the object of which the size is reduced inproportion to the distance that the indicator 535 is moved to the right(or the reduced depth value of the object 530). In response to a userinput 1120 moving the indicator 535 to the right again, the electronicdevice may display the object 530 of which the depth value and size arefurther reduced.

The electronic device may provide a visual effect in which theperspective of the object is adjusted according to the depth, byadjusting the size of the object while modifying the depth value of theobject using the UI representing the depth information.

FIG. 12 illustrates a sequence of operations of editing a plurality ofobjects in an image by using a UI representing depth information of theimage according to an embodiment of the disclosure.

Referring to FIG. 12, while the image 600 a and the object 530 aredisplayed, the electronic device may receive a user input 1210 forselecting another object 550. For example, the other object 550 mayinclude a sticker, a label, a drawing, a memo, an image, a mosaic, orcalligraphy. Although an embodiment of selecting another object 550different from the object 530 is illustrated in FIG. 12, the otherobject 550 may include an object of the same kind as the object 530.

According to an embodiment of the disclosure, in response to the userinput 1210, the electronic device may display the other object 550 andan indicator 555 representing the depth of another object 550. Accordingto an embodiment of the disclosure, the electronic device may displayanother object 550 having a predetermined depth value. For example, thepredetermined depth value may be the maximum depth value of the image600 a. In this case, the electronic device may display the indicator 555arranged at the leftmost side of the UI 505, and may display the entireanother object 550 in the image 600 a. The other object 550 may be seento the user as being arranged at the front-most position in the image600 a. According to an embodiment of the disclosure, to represent thatthe other object 550 is currently in the edit state, the electronicdevice may release the display of the indicator 540 representing theedit state of the object 530 and may display an indicator 560representing the edit state of the other object 550. According to anembodiment of the disclosure, the color or brightness of the indicator555 may be substantially the same as or at least partially differentfrom those of the indicator 535.

According to an embodiment of the disclosure, while the other object 550is displayed, the electronic device may receive a user input 1220 formoving the indicator 555. In response to the user input 1220, theelectronic device may display at least a part of the other object 550 ofwhich the depth is modified. For example, since the indicator 555 ismoved to the right (i.e., the direction in which the depth value isdecreased), the electronic device may determine the reduced depth valueof the other object 550 based on the distance that the indicator 555moves. When the reduced depth value of the other object 550 is less thanthe minimum depth value of the image region corresponding to theforeground subject 520 and greater than the depth value of the object530 (i.e., the indicator 555 is arranged between the depth range 515 andthe indicator 535 in the UI 505), the electronic device may display theother object 550 except for the region which overlaps the foregroundsubject 520. The other object 550 may be seen to the user as beinglocated in front of the background subject 510 and the object 530 andbehind the foreground subject 520.

The electronic device may easily adjust the depth of each of a pluralityof objects in an image by using a UI representing depth information.Although not illustrated in FIG. 12, the electronic device adjusts thesize of the other object 550 based on the same or similar principle asthat of the operations described in FIGS. 9 to 11, so that theelectronic device may provide a visual effect in which the perspectiveof each of the plurality of objects is adjusted.

FIGS. 13 to 15B illustrate a sequence of operations of selecting asubject in an image by using a UI representing depth information of theimage according to various embodiments of the disclosure. The operationsillustrated in FIG. 13 may be operations that more specificallyimplement operation 340 of FIG. 3.

Referring to FIG. 13, in operation 1310, the electronic device (e.g.,the first electronic device 101, the processor 120, or the image signalprocessor 260) may receive an input for selecting a depth range.

Referring to FIG. 14, the electronic device may display an image 1400 aand a UI 1405 representing depth information of the image 1400 a. Theimage 1400 a may include background subjects 1410 and 1420, and aforeground subject 1430. The UI 1405 may include a depth range 1415representing depth information of the image region corresponding to thebackground subject 1410, a depth range 1425 representing depthinformation of the image region corresponding to the background subject1420, and a depth range 1435 representing depth information of the imageregion corresponding to the foreground subject 1430. The colors orbrightness of the depth ranges 1415, 1425 and 1435 may substantially bethe same as, or at least partially different from each other based onthe depth information. The electronic device may receive a user inputfor moving an indicator 1445 for selecting a depth range. For example,as illustrated in FIG. 14, the electronic device may receive the userinput for moving the indicator 1445 such that the indicator 1445indicates the depth range 1435. Although an embodiment of selecting adepth range using two indicators is illustrated in FIG. 14, variousembodiments for selecting a depth range may be provided. For example,the user input for selecting a depth range may include at least one ormore tap operations, press operations, pressure touch operations,pinch-in operations, pinch-out operations, button operations, orcombinations thereof using a user's body (e.g., a finger) or an externaldevice (e.g., a touch pen).

In operation 1320, in response to the user input for selecting a depthrange, the electronic device may select an image region corresponding tothe depth range selected in the image. For example, referring to FIG.14, in response to the user input for moving the indicator 1445indicating the depth range 1435, the electronic device may select theimage area (e.g., the image region corresponding to the foregroundsubject 1430 in the image 1400 a) corresponding the depth range 1435.

In operation 1330, the electronic device may edit the selected imageregion. According to an embodiment of the disclosure, the electronicdevice may copy or remove the selected image region. For example,referring to FIG. 14, the electronic device may remove another imageregion except for the image region corresponding to the foregroundsubject 1430 from the image 1400 a. As another example, although notillustrated in FIG. 14, the electronic device may remove only the imageregion corresponding to the foreground subject 1430 from the image 1400a.

According to an embodiment of the disclosure, the electronic device maycombine the selected image region with another image.

FIG. 15A is a view illustrating an image composition using a UIrepresenting depth information of an image according to an embodiment ofthe disclosure.

Referring to FIG. 15A, while an image 1500 a and a UI 1505 aredisplayed, the electronic device may receive a user input for selectinga depth range 1535 in the UI 1505 and may select the image region (i.e.,the image region corresponding to the foreground subject 1530)corresponding to the selected depth range 1535. The electronic devicemay combine the selected image region with an image 1500 b. The image1500 b may include background subjects 1550 and 1560, and a foregroundsubject 1570. A UI 1545 representing the depth information of the image1500 b may include a depth range 1555 representing depth information ofan image region corresponding to the background subject 1550, a depthrange 1565 representing depth information of an image regioncorresponding to the background subject 1560, and a depth range 1575representing depth information of an image region corresponding to theforeground subject 1570. According to an embodiment of the disclosure,the electronic device may combine the image region corresponding to theforeground subject 1530 with the image 1500 b and may display the depthrange 1535 of the combined image region in the UI 1545. According to anembodiment of the disclosure, in response to the user input for movingthe depth range 1535, the electronic device may display the depth range1535 of which the location is changed and at least a part of the imageregion corresponding to the foreground subject 1530 of which the depthis modified.

Although an embodiment in which the depth range 1535 is added into theUI 1545 is illustrated in FIG. 15A, the electronic device may displaythe additional UI. Referring to FIG. 15B, the electronic device maydisplay the UIs 1545 and 1505. Although an example of arranging the UIs1545 and 1505 are arranged in the image 1500 b is illustrated in FIG.15B, the locations and forms of the UIs 1545 and 1505 are not limited tothose of FIG. 15B. For example, the UIs 1545 and 1505 may be arranged onthe screen below the image 1500 b. As another example, at least one ofthe UIs 1545 and 1505 may be displayed in a transparent form. As stillanother example, the UI 1505 may be arranged on a top end of the UI1545.

The electronic device may provide functions of more easily selecting andediting an image by selecting a specific image region in the image usinga UI representing a depth range.

FIG. 16 is a flowchart illustrating an electronic device for displayingan object of which a depth value is modified based on a user inputaccording to an embodiment of the disclosure.

Referring to FIG. 16, in operation 1610, the electronic device (e.g.,the first electronic device 101, the processor 120, or the image signalprocessor 260) may display an image including at least one subjectthrough the display device (e.g., the display device 160). For example,the image may include an image captured through the camera sensor of theelectronic device or a preview image.

In operation 1620, the electronic device may display the object withinthe displayed image. The object may include, for example, a sticker, alabel, a drawing, a memo, an image, a mosaic, or calligraphy. Forexample, an electronic device may display an object in response toreceiving a user input for inserting an object in the image.

In operation 1630, the electronic device may modify the depth value ofthe object based on the received user input. The user input may includevarious types of user inputs for modifying the depth of the displayedobject. For example, the user input may include a tap operation, a pressoperation, a pressure touch operation, a hovering operation, a panningoperation, a swipe operation, a flick operation, a hovering operation, abutton input operation, a drag operation, or a combination thereof,using a user's body or an external device.

In operation 1640, the electronic device may display at least a part ofthe object of which the depth is modified in response to receiving theuser input.

FIGS. 17A to 17C are views illustrating a sequence of operations ofdisplaying an object of which a depth value is modified based on a userinput according to various embodiments of the disclosure.

According to an embodiment of the disclosure, the electronic device maymodify the depth value of the object based on the depth information orthe color information of the region where the object moved based on theuser input for moving the object is located.

Referring to FIG. 17A, while an image 1710 a into which an object 1714is inserted is displayed, the electronic device may receive a user input1716 a or 1716 b (e.g., a swipe action, a panning operation, a dragoperation, or a flick operation) for moving the object 1714 in aspecific direction. In response to the user input 1716 a, the electronicdevice may change the depth value of object 1714 based on the depth orcolor information of the region in which the moved object 1714 islocated. For example, when the depth values or RGB values of a certainnumber or more of pixels among pixels in the region in which the movedobject 1714 is located are the same as or similar to those of the pixelsconstituting a subject 1712, the electronic device may display theentire object 1714 in the image 1710 a. The object 1714 may be seen tothe user as being arranged in front of the subject 1712. As anotherexample, when the depth values or RGB values of a certain number or moreof pixels among pixels in the region in which the moved object 1714 islocated are the same as or similar to those of the pixels constituting abackground subject, the electronic device may display the object 1714except for the region that overlaps the subject 1712 in the image 1710a. The object 1714 may be seen to the user as being arranged behind thesubject 1712.

According to an embodiment of the disclosure, the electronic device maymodify the depth value of the object in response to a user input formodifying the depth value of the object by operating a button. Referringto FIG. 17B, while an image 1720 a in which an object 1724 is arrangedin front of a subject 1722 is displayed, the electronic device mayreceive a user input for selecting a button 1726 to reduce the depthvalue of the object 1724. The electronic device may modify the depthvalue of the object 1724 to be less than the minimum depth value of theimage region corresponding to the subject 1722 in response to a userinput for selecting the button 1726. As another example, while theobject 1724 is displayed behind the subject 1722, in response to a userinput for selecting a button 1728 to increase the depth value of theobject 1724, the electronic device may modify the depth value of theobject 1724 to be greater than the maximum depth value of the imageregion corresponding to the subject 1722.

According to another embodiment of the disclosure, the electronic devicemay modify the depth value of the object in a preview image. Referringto FIG. 17C, while a preview image 1730 a is displayed, the electronicdevice may receive a user input 1736 for auto-focusing a subject 1732.The electronic device may increase the depth value of the subject 1732or decrease the depth value of an object 1734 in response to the userinput 1736. While the object 1734 is displayed behind the subject 1732,the electronic device may increase the depth value of the subject 1732in response to the user input for selecting a button 1738.

The electronic device may provide convenience to the user by displayingan object having a modified depth value in response to various userinputs in addition to a scheme using a UI representing depthinformation.

As described above, the electronic device (e.g., the first electronicdevice 101) disclosed in the disclosure may include a display unit(e.g., the display device 160), a memory (e.g., the memory 130) forstoring instructions, and a processor (e.g., the processor 120 or theimage signal processor 260), where the processor may execute the storedinstructions to display the image and a user interface (e.g., the UI505) representing the depth information of the image through the displayunit in response to a first user input for editing the image (e.g., theimage 600 a), receive a second user input for selecting an object (e.g.,the object 530) to be added in the image, display at least a part of theobject in the image based on a depth value of the object and depthinformation of a first region in which the object is located in theimage, and display an indicator (e.g., the indicator 535) representingthe depth value of the object in the UI. According to an embodiment ofthe disclosure, the UI may be displayed in the image or on a lower endof the image.

According to an embodiment of the disclosure, the processor may executethe stored instructions to display the object having a predetermineddepth value in the first region in response to the second user input,modify the predetermined depth value of the object in response to athird user input (e.g., the user input 620) for moving the indicator,and display at least a part of the object in the image based on themodified depth value of the object and the depth information of thefirst region.

According to an embodiment of the disclosure, the processor may executethe stored instructions to determine a size of the object based on themodified depth value of the object, and display at least a part of theobject in the image based on the modified depth value of the object, thedepth information of the first region and the size of the object.

According to an embodiment of the disclosure, the processor may executethe stored instructions to receive a fourth user input (e.g., thebackground image 600 b) for selecting a background image to be added inthe image, and display at least a part of the background image in theimage based on a depth value of the background image and the depthinformation of the image.

According to an embodiment of the disclosure, the processor may executethe stored instructions to receive a fifth user input for selecting adepth range in the UI, and display a second region corresponding to theselected depth range in the image. In addition, the processor mayexecute the stored instructions to combine an image of the second regionwith another image.

According to an embodiment of the disclosure, the processor may executethe stored instructions to display a second indicator (e.g., theindicator 540) representing an edited state of the object in response tothe second user input, and release the display of the second indicatorin response to a sixth user input (e.g., the user input 710) forselecting a region except the first region in the image.

As described above, a method of an electronic device disclosed in thedisclosure may include an operation of displaying an image (e.g., theimage 600 a) a user interface (e.g., the UI 505) representing and depthinformation of the image in response to a first user input for editingthe image, an operation of receiving a second user input for selectingan object (e.g., the object 530) to be added in the image, an operationof displaying at least a part of the object in the image based on adepth value of the object and depth information of a first region inwhich the object is located in the image, and an operation of displayingan indicator (e.g., the indicator 535) representing the depth value ofthe object in the UI.

According to an embodiment of the disclosure, the method may furtherinclude an operation of displaying the object having a predetermineddepth value in the first region in response to the second user input, anoperation of modifying the predetermined depth value of the object inresponse to a third user input (e.g., the user input 620) for moving theindicator, and an operation of displaying at least a part of the objectin the image based on the modified depth value of the object and thedepth information of the first region.

According to an embodiment of the disclosure, the method may furtherinclude an operation of determining a size of the object based on themodified depth value of the object, and an operation of displaying atleast a part of the object in the image based on the modified depthvalue of the object, the depth information of the first region and thesize of the object.

According to an embodiment of the disclosure, the method may furtherinclude an operation of receiving a fourth user input for selecting abackground image (e.g., the background image 600 b) to be added in theimage, and an operation of displaying at least a part of the backgroundimage in the image based on a depth value of the background image andthe depth information of the image.

According to an embodiment of the disclosure, the method may furtherinclude an operation of receiving a fifth user input for selecting adepth range in the UI, and an operation of displaying a second regioncorresponding to the selected depth range in the image. In addition, themethod may further include an operation of combining an image of thesecond region with another image.

According to an embodiment of the disclosure, the method may furtherinclude an operation of displaying a second indicator representing anedited state of the object in response to the second user input, and anoperation of releasing the display of the second indicator in responseto a sixth user input for selecting a region except the first region inthe image.

As described above, the electronic device (e.g., the first electronicdevice 101) disclosed in the disclosure may include at least one imagesensor (e.g., the camera module 188 or the image sensor 230), a displayunit (e.g., the display device 160), a memory (e.g., the memory 130) forstoring instructions, and a processor (e.g., the processor or the imagesignal processor 260), where the processor may execute the storedinstructions to obtain a color image (e.g., the color image 500 a) and adepth map (e.g., the depth map 500 b) corresponding to the color imagethrough the at least one image sensor, generate an image (e.g., theimage 600 a) by combining the color image with the depth map, displaythe image and a user interface (e.g., the UI 505) representing depthinformation of the image through the display unit in response to a firstuser input for editing the image, receive a second user input forselecting an object (e.g., the object 530) to be added in the image,display at least a part of the object in the image based on a depthvalue of the object and depth information of a first region in which theobject is located in the image, and display an indicator (e.g., theindicator 535) representing the depth value of the object in the UI.

According to an embodiment of the disclosure, the processor may executethe stored instructions to display the object having a predetermineddepth value in the first region in response to the second user input,modify the predetermined depth value of the object in response to athird user input (e.g., the user input 620) for moving the indicator,and display at least a part of the object in the image based on themodified depth value of the object and the depth information of thefirst region.

According to an embodiment of the disclosure, the processor may executethe stored instructions to determine a size of the object based on themodified depth value of the object, and display at least a part of theobject in the image based on the modified depth value of the object, thedepth information of the first region and the size of the object.

According to an embodiment of the disclosure, the processor may executethe stored instructions to receive a fourth user input for selecting abackground image (e.g., the background image 600 b) to be added in theimage, and display at least a part of the background image in the imagebased on a depth value of the background image and the depth informationof the image.

According to an embodiment of the disclosure, the processor may executethe stored instructions to receive a fifth user input for selecting adepth range in the UI, and display a second region corresponding to theselected depth range in the image.

The electronic device according to various embodiments disclosed in thedisclosure may be various types of devices. The electronic device mayinclude, for example, at least one of a portable communication device(e.g., a smartphone), a computer device, a portable multimedia device, amobile medical appliance, a camera, a wearable device, or a homeappliance. The electronic device according to an embodiment of thedisclosure should not be limited to the above-mentioned devices.

It should be understood that various embodiments of the disclosure andterms used in the embodiments do not intend to limit technologiesdisclosed in the disclosure to the particular forms disclosed herein;rather, the disclosure should be construed to cover variousmodifications, equivalents, and/or alternatives of embodiments of thedisclosure. With regard to description of drawings, similar componentsmay be assigned with similar reference numerals. As used herein,singular forms may include plural forms as well unless the contextclearly indicates otherwise. In the disclosure disclosed herein, theexpressions “A or B”, “at least one of A or/and B”, “A, B, or C” or “oneor more of A, B, or/and C”, and the like used herein may include any andall combinations of one or more of the associated listed items. Theexpressions “a first”, “a second”, “the first”, or “the second”, used inherein, may refer to various components regardless of the order and/orthe importance, but do not limit the corresponding components. The aboveexpressions are used merely for the purpose of distinguishing acomponent from the other components. It should be understood that when acomponent (e.g., a first component) is referred to as being (operativelyor communicatively) “connected,” or “coupled,” to another component(e.g., a second component), it may be directly connected or coupleddirectly to the other component or any other component (e.g., a thirdcomponent) may be interposed between them.

The term “module” used herein may represent, for example, a unitincluding one or more combinations of hardware, software and firmware.The term “module” may be interchangeably used with the terms “logic”,“logical block”, “part” and “circuit”. The “module” may be a minimumunit of an integrated part or may be a part thereof. The “module” may bea minimum unit for performing one or more functions or a part thereof.For example, the “module” may include an application-specific integratedcircuit (ASIC).

Various embodiments of the disclosure may be implemented by software(e.g., the program 140) including an instruction stored in amachine-readable storage media (e.g., an internal memory 136 or anexternal memory 138) readable by a machine (e.g., a computer). Themachine may be a device that calls the instruction from themachine-readable storage media and operates depending on the calledinstruction and may include the electronic device (e.g., the firstelectronic device 101). When the instruction is executed by theprocessor (e.g., the processor 120), the processor may perform afunction corresponding to the instruction directly or using othercomponents under the control of the processor. The instruction mayinclude a code generated or executed by a compiler or an interpreter.The machine-readable storage media may be provided in the form ofnon-transitory storage media. Here, the term “non-transitory”, as usedherein, is a limitation of the medium itself (i.e., tangible, not asignal) as opposed to a limitation on data storage persistency.

According to an embodiment of the disclosure, the method according tovarious embodiments disclosed in the disclosure may be provided as apart of a computer program product. The computer program product may betraded between a seller and a buyer as a product. The computer programproduct may be distributed in the form of machine-readable storagemedium (e.g., a compact disc read only memory (CD-ROM)) or may bedistributed only through an application store (e.g., a Play Store™). Inthe case of online distribution, at least a portion of the computerprogram product may be temporarily stored or generated in a storagemedium, such as a memory of a manufacturer's server, an applicationstore's server, or a relay server.

Each component (e.g., the module or the program) according to variousembodiments may include at least one of the above components, and aportion of the above sub-components may be omitted, or additional othersub-components may be further included. Alternatively or additionally,some components (e.g., the module or the program) may be integrated inone component and may perform the same or similar functions performed byeach corresponding component prior to the integration. Operationsperformed by a module, a programming, or other components according tovarious embodiments of the disclosure may be executed sequentially, inparallel, repeatedly, or in a heuristic method. In addition, at leastsome operations may be executed in different sequences, omitted, orother operations may be added.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. An electronic device comprising: at least oneimage sensor; a display; a processor; and a memory configured to storeinstructions, wherein the instructions, when executed by the processor,cause the processor to: obtain, via the at least one image sensor, acolor image and a depth map corresponding to the color image, generatean image by combining the color image with the depth map, control thedisplay to display the image, while displaying the image, receive afirst user input selecting an object to be added to the image, and inresponse to receiving the first user input, control the display todisplay at least a part of the object in the image, based on a depthvalue of the object and depth information of a first region in which theobject is located in the image, and wherein the depth information of theimage is determined based on the depth map before receiving the firstuser input.
 2. The electronic device of claim 1, wherein theinstructions further cause the processor to: in response to receivingthe first user input, control the display to display a user interface(UI) representing the depth information of the image, and control thedisplay to display a first indicator representing the depth value of theobject in the UI.
 3. The electronic device of claim 2, wherein theinstructions further cause the processor to: in response to receivingthe first user input, control the display to display the object having apredetermined depth value in the first region, in response to receivinga second user input for moving the first indicator, modify thepredetermined depth value of the object, and control the display todisplay the part of the object in the image based on the modified depthvalue of the object and the depth information of the first region. 4.The electronic device of claim 3, wherein the instructions further causethe processor to: determine a size of the object based on the modifieddepth value of the object, and control the display to display the partof the object in the image based on the modified depth value of theobject, the depth information of the first region, and the size of theobject.
 5. The electronic device of claim 1, wherein the instructionsfurther cause the processor to: receive a third user input selecting abackground image to be added to the image, and control the display todisplay at least a part of the background image in the image based on adepth value of the background image and the depth information of theimage.
 6. The electronic device of claim 2, wherein the instructionsfurther cause the processor to: receive a fourth user input selecting adepth range in the UI, and control the display to display a secondregion corresponding to the selected depth range in the image.
 7. Theelectronic device of claim 6, wherein the instructions further cause theprocessor to combine an image of the second region with another image.8. The electronic device of claim 1, wherein the instructions furthercause the processor to: in response to receiving the first user input,control the display to display a second indicator representing an editedstate of the object, and in response to receiving a fifth user inputselecting a region except the first region in the image, release thedisplay of the second indicator.
 9. A method of an electronic device,the method comprising: obtaining, via at least one image sensor, a colorimage and a depth map corresponding to the color image; generating animage by combining the color image with the depth map; displaying theimage on a display of the electronic device; while displaying the image,receiving a first user input selecting an object to be added to theimage; and in response to receiving the first user input, displaying atleast a part of the object in the image, based on a depth value of theobject and depth information of a first region in which the object islocated in the image, wherein the depth information of the image isdetermined based on the depth map before receiving the first user input.10. The method of claim 9, further comprising: in response to receivingthe first user input, displaying a user interface (UI) representing thedepth information of the image, and displaying a first indicatorrepresenting the depth value of the object in the UI.
 11. The method ofclaim 10, further comprising: in response to receiving the first userinput, displaying the object having a predetermined depth value in thefirst region; in response to receiving a second user input for movingthe first indicator, modifying the predetermined depth value of theobject; and displaying the part of the object in the image based on themodified depth value of the object and the depth information of thefirst region.
 12. The method of claim 11, further comprising:determining a size of the object based on the modified depth value ofthe object; and displaying the part of the object in the image based onthe modified depth value of the object, the depth information of thefirst region, and the size of the object.
 13. The method of claim 9,further comprising: receiving a third user input selecting a backgroundimage to be added to the image; and displaying at least a part of thebackground image in the image based on a depth value of the backgroundimage and the depth information of the image.
 14. The method of claim10, further comprising: receiving a fourth user input selecting a depthrange in the UI; and displaying a second region corresponding to theselected depth range in the image.
 15. The method of claim 14, furthercomprising combining an image of the second region with another image.16. The method of claim 9, further comprising: in response to receivingthe first user input, displaying a second indicator representing anedited state of the object; and in response to receiving a fifth userinput selecting a region except the first region in the image, releasingthe display of the second indicator.